Skip to content

Add support for list attributes in metadata#36

Open
pau-mensa wants to merge 2 commits into
lightonai:mainfrom
pau-mensa:main
Open

Add support for list attributes in metadata#36
pau-mensa wants to merge 2 commits into
lightonai:mainfrom
pau-mensa:main

Conversation

@pau-mensa

Copy link
Copy Markdown

I've had this use case for quite a while now, where I've wanted to store list attributes in the metadata of documents. Since sqlite does not support list attributes I stored them as strings and managed my way around using LIKE and OR, but this got to a point where it does not scale further.

I implemented this solution that creates a metadata_inverted.json per index, where, per each list attribute, the distinct elements as keys and the document ids where they are referenced as values are stored. During filtering time there are two new methods where_any and where_all that perform set-like operations on those list attributes.

If you are interested in merging this I can tweak the PR and adapt it however you want, on the other hand, if you are not interested in it (too much code added, don't like the solution/implementation, conflicts with other stuff and I'm missing context) feel free to close it!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant